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PREFACE 


This report has been issued in two volumes. IDA Paper 
P-1537, Part A, contains detailed background information on 
DROLS operations and is primarily intended for those readers 
who are not so familiar with DROLS. This volume. Part B, dis¬ 
cusses the research performed by IDA in response to the task 
statement. 
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I. INTRODUCTION 


The Defense Technical Information Center (DTIC), under the 
overall direction of the Office of the Deputy Under Secretary 
of Defense (Research and Advanced Technology), OUSDRE, operates 
a DoD-wide on-line system that provides remote interaction with 
four major RDT&E data bases. The system was originally designed 
in the late 1960s to provide secure access to a computerized file 
containing classified descriptions of ongoing research efforts. 
This original file of Work Unit Information System data was 
set up as a management information system (MIS) intended to help 
keep DoD managers informed about ongoing DoD scientific and tech¬ 
nical projects. Such a system would help to avoid duplication 
of technical effort, help managers and research personnel to 
stay knowledgeable about rapidly advancing science and technology 
and enable them to borrow and use ideas from each other that might 
be relevant to their projects. 

Once the system was in place and operating, it became ap¬ 
parent that it could have a much broader application than the 
one for which it was expressly designed. Three data bases were 
added: The Technical Reports (TR), Research and Development 

Program Planning (PP), and Independent Research and Development 
(IR&D) files. The operational system became known as the Defense 
RDT&E On-Line System (DROLS). 

Of the four files the TR file is by far the largest, con¬ 
taining about 1.3 million entries. Also, as might be expected, 
the TR file is the most frequently queried. Although it was 
designed and constrained to protect a small proportion (about 
10 percent of intrinsically classified records, DROLS is now 


very different in scope and intent from its original implemen¬ 
tation with the work unit file. Because of what seem, from this 
time vantage point, to be arbitrary design decisions, DROLS is 
not compatible with any other DoD systems for electronic access 
to or transfer of S&T information. DTIC cannot, for instance, 
interface with and utilize the computer resources sharing and 
terminal access capabilities of the DoD-developed ARPANET. It 
has difficulty taking advantage of the many software and hard¬ 
ware developments in the ADP marketplace. These deficiencies 
are manifested by high costs for operation, access and enhance¬ 
ment . 


DTIC has found some ways of circumventing the original 
limitations of DROLS. This year, for example, it has developed 
software that permits unclassified users to access unclassified 
fields that pertain to classified reports in the file. In ad¬ 
dition, it is now possible for low-speed asynchronous dial-up 
terminals to access the unclassified DROLS files. Since this 
study was started, these and other improvements have been made 
to DROLS that effectively enhance unclassified users’ capability 
to obtain almost as much information from the files as the 
cleared users. We believe that DTIC has gone about as far as it 
can in enhancing the utility of the unclassified terminals while 
still conforming to its mandate to protect classified information 
by releasing it only to those who are properly cleared to receive 
it. It has been able to accomplish what is has done so far, 
partly because only a small portion of the information in its 
files is classified and even of those reports that are clas¬ 
sified, a number of fields such as title, AD number, authors and 
descriptors are not classified. 

However, the protection of a small amount of classified 
information complicates further improvement of the system. It 
also maintains the high operating costs associated with secure 
operating systems. 


ft 





In October 1979, during a ceremony commemorating the re¬ 
designation of the Defense Documentation Center to Defense Tech¬ 
nical Information Center (DTIC), Dr. Ruth M. Davis, then the 
Deputy Under Secretary of Defense for Research and Engineering 
(R&AT), addressed DTIC on the subject of its role. Dr. Davis 
said that DTIC should expect to "work more closely with R&D 
managers in the Pentagon and provide them with technical and 
management information." In addition, she recommended that DTIC 
"play a more important role in the evolution of OUSDRE data pro¬ 
cessing activities ... and assisting and improving our management 
information systems as well." DTIC should "become part of a 
community providing internal and external interactive data pro¬ 
cessing support to OSD." It should have "R&D capability to take 
care of its own growing needs and the requirements of the Defense 
STI program." DTIC in the future will "contribute to the edu¬ 
cation and training of R&D managers in the use of DTIC and other 
data bases and in generating a Federal-wide information inter¬ 
change program. It will also encourage the improvement of the 
quality of technical reports..." (Ref. 1). 

In view of these changes in scope and intent, it seems rea¬ 
sonable to examine the purposes and priorities of DTIC insofar 
as they concern the handling of classified information on-line. 
For example, considering the latest developments in information 
technology, would there be a net advantage to DTIC and the Com¬ 
munity it serves, as well as the Department of Defense, if DROLS 
became a completely unclassified system? What alternatives are 
available to DTIC to supply the needs of the community that uses 
classified information, if DROLS were limited to unclassified 
information? What would be users attitude toward such a change? 
What alternative courses of action do users have if they can no 
longer obtain classified citations from DTIC on-line? 
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In this study we have tried to examine these questions. We 
have had to remind ourselves continually that DROLS does not have 
in its banks much that would be considered by scientists and en¬ 
gineers as "real" data. DROLS provides primarily bibliographic 
information and management information such as who is working on 
what. DROLS then is really more a source of where to find it 
rather than of information and data. It is not the only such 
source that DTIC provides, but it is the most convenient, time 
saving and comprehensive source of where to find it information 
that DTIC offers. Unfortunately, although only a small part of 
this information is classified, it is necessary that the entire 
system be set up to protect this small proportion of classified 
information. Hence, all information that goes out to classified 
terminals must be encrypted and all such terminals must be lo¬ 
cated in appropriate enclosures and attended by properly cleared 
personnel. In addition, strict controls must be maintained on 
user protocols for addressing the system to ensure that users 
receive only the information they are permitted to receive by 
virtue of their security status and need to know. 

DTIC asked IDA to provide an examination and analysis that 
might help to improve access to DROLS. Of particular interest 
was the question of whether DROLS should become a totally un¬ 
classified system, with classified user's needs handled by some 
other means. To examine this question IDA was asked to examine 
the basic objectives and priorities for providing on-line access 
to and communications of relevant DTIC-stored Scientific and 
Technical Information (STI) in support of Defense RDT&E programs 
and also the National science and technology community. 

This report describes the work and results of this IDA 
effort, which was initiated on December 10, 1979 * During the 
course of this work, DTIC initiated a number of enhancements of 
DROLS that improved considerably the service to the unclassified 
user. Some of these improvements came unexpectedly, and, since 
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some of the subtasks were made obsolete by the improvements, the 
study task was redirected by mutual agreement. In this report 
we have attempted to indicate the work that was accomplished in 
response to the original task statement as well as to the re¬ 
direction. 
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II. SUMMARY AND CONCLUSIONS 

In December 1979, the Defense Technical Information Center 
(DTIC) requested the Institute for Defense Analyses (IDA) to 
undertake a study of its remote terminal accessible bibliographic 
reference system known as the Defense RDT&E On-Line System (DROLS). 
The requested study was to consider the principal objectives and 
priorities involved in the operation and design of on-line scien¬ 
tific and technological data base systems in support of the De¬ 
fense RDT&E community, to review the current secure communications 
service used by DROLS, to survey a sample of DROLS users to obtain 
their reactions to the proposition that DROLS become an unclassi¬ 
fied system, to examine the need and utility of the classified 
information available from DROLS and to provide some cost infor¬ 
mation relative to classified and unclassified terminals. 

Our examination of the applicable regulations, directives 
and instructions revealed a long list of documents whose intention 
is to make DTIC a central repository of Defense RDT&E information. 
The regulations also charge DTIC with the requirement to protect 
all such information that may be classified. There are some 
limits to the extent of DTIC's domain. For example, it is not 
authorized to have any information classified above the "Secret" 
level, nor is it to have any communications security or intelli¬ 
gence information in its files. In addition, DTIC is required to 
limit access to any information that might be classed as limited 
distribution by anyone, although, insofar as DROLS is concerned, 
this only prevents some users from seeing abstracts of such 
documents. The most significant limitation to DTIC's services 
and to DROLS is derived from another problem—documents that 
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should be submitted are simply withheld; thus, there Is no in¬ 
dication anywhere in the files that such documents exist. We 
found estimates that as much as fifty percent of all documents 
that should be submitted to DTIC are withheld, thus severely 
hampering the effectiveness of DTIC and of DROLS in performing 
their assigned functions. 

In the past year DTIC, in making improvements to DROLS, has 
made it possible to access the system and receive unclassified 
information from asynchronous dial-up terminals anywhere that 
they can connect with a telephone line. Furthermore, the system 
has been changed so that citations will be obtained to almost 
all reports in the file even on an unclassified terminal. Thus, 
the unclassified terminal will receive nearly as much information 
as will the classified terminal. This latter fact means little 
to most of the DROLS users who have classified terminals, who 
show no interest in the possible savings and better service to 
unclassified terminal users that might result from converting 
DROLS to an unclassified system. The prevailing attitude of 
classified terminal users is that the information and time saved 
in obtaining it on-line are well worth whatever it costs. 

Further, we examined four possible alternatives that DTIC 
might follow in dealing with DROLS. It is our view that the best 
alternative for now would be to keep DROLS substantially as it is 
a simultaneous classified and unclassified system. The basic 
reason for this conclusion is that the classified terminal users 
would object strenuously to such a change even though an unclassi 
fied system seems to be able to meet virtually all of their needs 
at lower cost to DTIC, all users and to the DoD. 

The only disadvantage of not having a classified DROLS is 
that a small proportion of citations might be missing or less 
than complete. Nevertheless, the classified user could order 
the classified citations to be delivered by mail. Alternatively, 
if the user were pressed for time, he could order the reports 
themselves with some greater risk of ordering something that did 


not meet his requirements. We found that since all authorized 
DTIC information would be available to classified users by mail 
or other means, though some small part of it may be delayed in 
coming if DROLS were unclassified, the problem boils down to the 
value of timeliness in obtaining information, a value that we 
feel we could not credibly determine. 


More detailed findings and conclusions are presented in 
Chapter VI. 
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III. SOME BACKGROUND CONSIDERATIONS 


A. PRIOR LITERATURE 

There is a considerable body of literature that describes 
the developing situation in information science; but not all of 
it is relevant to the particular study we have made for DTIC. 
Rather a considerable proportion is devoted to the historical 
development of the field and some of it to policy development. 
This is especially true of Ref. 2, which deals with the problems 
of scientific and technical communication in both the context 
of the United States and on a worldwide basis. Another treat¬ 
ment in this genre attempts to forecast the situation in 1985 
(Ref. 3)» Reference 4, on the other hand, is more focused on 
the problems of DTIC. This study was done in 1975 for DDC by 
Auerbach Associates and is labeled as a 10-year Requirements 
and Planning Study. The work, which is based almost entirely 
on a large survey of DDC users, emphasizes the user's point of 
view in developing its conclusions. This study was focused on 
all DDC services, whereas the current study is directed only to 
DROLS. 

One finding of the Auerbach report was that users prefer 
to deal with their local library rather than remote libraries. 
This finding encouraged DDC to develop DROLS as a service that 
is available to the local librarian rather than directly to 
the ultimate user at his desk or bench. DDC developed train¬ 
ing programs to teach librarians how to use DROLS. According 
to the Auerbach report most users are unaware of the full 
range of DDC services. This has encouraged DTIC to engage in 
some low profile marketing of its services with the result that 
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many librarians are now familiar with DTIC services even 
though many bench-level researchers and managers could use 
more instruction concerning what information support is 
available. 

One of the Auerbach report's more significant conclusions 
was that DDC should move to providing fact and information ser¬ 
vices as well as bibliographic and document-oriented services. 
This same kind of recommendation appeared in our limited sur¬ 
vey of DTIC users. Thus far the holdings of only five Informa¬ 
tion Analysis Centers (IAC) can be identified on-line by DTIC 
users. Other IACs must be contacted directly by the user with¬ 
out going through DTIC. To change this, new DoD regulations are 
needed that will permit DTIC to expand into new data files. A 
recent DTIC Digest indicates that they are making plans to move 
in this direction. In addition, the Shared Bibliographic Input 
Experiment (SBIE) potentially will build a Defense On-Line Cat¬ 
alog at DTIC which also will serve as a referral clearinghouse 
for documents and data files from sources other than DTIC within 
the Defense Community. 

B. THE AUERBACH SURVEY OF INSTITUTIONS 

Part of the Auerbach Associates' study for DDC, the "DDC 
10-year Requirements and Planning Study," involved a survey of 
fourteen U.S. Government agencies. This survey was called the 
Interagency Survey Report (Ref. 5) and its purpose was to de¬ 
velop a projection of goals, objectives, operations, services, 
and structure of the information environment that DDC will be 
facing through 1988. The findings were the following: 

1. DDC’s primary role is to serve the DoD RDT&E 
community. 

2. DDC’s secondary role is to serve Federal agencies. 

3. DDC's tertiary role is to advance the state of 
the art of information processing in the nation. 


The study points up the fact that technological problems 
of information processing are not as significant in defining 
the scope and nature of services as are the organizational 
and economic ones. Also, the available technology has not yet 
been exploited fully by the information community. The report 
urges that the parochial interests of military management 
information programs should give way to achieve implementation 
of effective RDT&E management information systems. They admit 
that no value measures of information have been developed that 
would be useful in making economic decisions. Rather, informa¬ 
tion has subjective and pragmatic value to its users, but 
there are no measures of this value. This finding is consis¬ 
tent with opinions and attitudes we found in our own user 
survey. The Auerbach report further urged DDC to develop and 
apply econometric measures to information processes and to 
increase its efforts to market DDC products and services. 

The Auerbach study also suggests that DDC develop new 
products to satisfy a wider range of DoD information needs. 

For example, an effective RDT&E Management Information System 
would help RDT&E managers assess their options at decision 
points in their programs. 

C. THE AINES REPORT 

In January, 1978, a report (Ref. 6) was published by OUSDRE 
(Research and Advanced Technology). The author, Andrew A. Aines, 
was a senior staff associate of the National Science Foundation 
on loan to OUSDRE (R&AT). The purpose of the study was to 
appraise the health of the DoD Scientific and Technical Infor¬ 
mation (STI) programs. It was intended to result in improvement 
and growth of these programs. 

This study was undertaken in response to the rapidly 
emerging large-scale use of advanced information technology, 
in recognition of the need for improved effectiveness of the 
many DoD information systems and also in recognition of the 


fact that more than ten years had passed since the last formal 
study of DoD STI programs. Other factors were the enormous 
proliferation of information and recognition of the importance 
of information as a lever of power. The study was intended to 
suggest improvements that would be useful and compatible with 
how DoD STI programs should look in the mid-1980s time period. 
The study involved surveys and interviews with research and 
engineering managers as well as managers of DoD technical 
information programs. 

Aines concluded that although the DoD STI program was ac¬ 
tive and operative it had a number of weaknesses, particularly 
in the area of management, that needed to be rectified. The 
management problems identified included the following: 

1. Directives and regulations were outdated. 

2. STI had low priorities in funding and in 
command attention. 

3. Reorganizations were too frequent, personnel 
turnover was too high, and the status of 
information managers was too low. 

4. Interaction between R&E managers and 
information managers was too low. 

5. Data bases were not being used effectively. 

6. The use of advanced information technology 
was not moving evenly or smoothly. 

7. The benefits of a government-wide STI system 
featuring electronic interchange were not 
receiving enough attention. 


Despite these management problems, DDC was probably 
(and still is) the most advanced information agency in the 
federal government. However, it was not using its capabilities 
fully to aid in improving National policies and programs. 
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Aines recommended various measures to offset these mana¬ 
gerial problems. He urged that long- and mid-range plans be 
developed to relate OUSDRE, the military services and the other 
Defense agencies. A DoD coordinating council should be formed. 

A task group should be created to revise outdated STI directives 
and regulations and annual reviews of STI should be undertaken 
in OUSDRE and other Defense agencies. 

Relationships between information agencies and other 
kinds of organizations such as professional societies, educa¬ 
tional institutions and other governmental units should be 
studied and improved. Supervision of STI programs should be 
improved and collaboration among the STI, logistical and 
intelligence communities should be fostered. DDC should be 
upgraded to provide better support for R&D managers and re¬ 
search workers, and as a sign of this upgrading, its name 
should be changed to Defense Technical Information Center— 
a suggestion that has since be-en adopted. 


IV. USER ATTITUDES TOWARD DROLS CLASSIFICATION 

A. INTRODUCTION 

Under the revised subtask b (see Appendix A) IDA was 
requested to survey a sample of DROLS users to obtain their 
reactions to the proposition that DROLS become a completely 
unclassified system. The amended subtask c was also dealt with 
by analyzing the survey. This chapter describes the survey 
and presents the results and conclusions therefrom. 

B. METHOD OF STUDY 

One approach to this study involved a questionnaire (see 
Appendix C) which was devised to be easy to use and convenient 
for both the respondent and the surveyor. Originally we had in¬ 
tended to survey all users of DROLS (about 100) but then we were 
informed that such a survey would involve severe problems in ob¬ 
taining permission. However, we could interview people at up to 
about ten sites, plus personnel at IDA, without problems. So we 
decided that we would go to eight major users in the Washington, 
D.C., area and two outside. This would avoid incurring heavy 
travel costs. Since it was impossible to do a statistically 
valid survey with only eleven respondents, we felt the best we 
could do was to try to get some insights about user attitudes 
with this number of respondents. Nevertheless, we did try for 
a distribution of types of users to ensure that whatever opin¬ 
ions we did get would be reasonably representative. We used 
the following matrix (Exhibit IV-1) as a guide, and tried to 
provide a distribution of respondents as shown throughout the 
matrix. 
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EXHIBIT IV-1 


DISTRIBUTION OF SURVEY RESPONDENTS 
(U = Unclassified Terminal, C = Classified Terminal) 



LIBRARIES 
& SCHOOLS 

R&D 

LABORATORY 

OPERATING 

COMMAND 

Army 

Army 

Library (U )(C) 

BRL (C) 


Navy 


NSWC (C) 

NRL (C) 

NAVMAT (C) 

Air Force 

AFOSR (U) 

AFAL (C) 


DOD 


DARPA (U) 

DIA (C) 

Contractor 


IDA (C) 

Lincoln Lab (C) 



The method used in setting up the interviews was to tele¬ 
phone the individual, usually a librarian or information source # 

person, listed as the DTIC contact in a DTIC listing of termi¬ 
nal sites. This individual was told what we were planning and 
that he or she would shortly receive a packet containing some 
descriptive material and several questionnaires. After they • 

had time to examine the material, they were recontacted and an 
appointment for interviews was set up. Each contact was asked 
to line up two other individuals for interviews—one a resear¬ 
cher, the other a research manager. We had two reasons for this • 

elaborate procedure. First, we felt that it would be more pro¬ 
ductive if people had an opportunity to read over the material 
and questionnaires before responding, particularly since some of 
the questions might involve finding some data. Second, we felt • 
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that although seemingly less efficient than the use of a mailed 
questionnaire, the face-to-face interview would enable us to 
learn more and to detect nuances of intended meaning because we 
could probe somewhat further if the answers to some questions 
(or if the questions themselves) seemed inappropriate. Further¬ 
more, we would be there to explain, as necessary, any or all of 
the questions. 

As it turned out, this mode of conducting the survey was 
very fruitful. Nearly all individuals we surveyed had read 
the material beforehand and had marked up their questionnaires. 
Where cost and other data were requested they had taken the 
trouble to look up the information. The interviews themselves 
generally went very smoothly and quickly since the primary task 
was to transfer information from their questionnaires to ours. 

In most cases, the longest part of the interview involved a dis¬ 
cussion of question 16, where the subjects were asked for their 
ideas on improvements to DROLS, or involved a general discussion 
of how each individual used on-line information in his work. 

The questionnaire and introductory material that was sent 
out are reproduced in Appendix C. The findings are based on 
responses of 28 individuals at 11 organizations. 

C. WHAT WE LEARNED 

What we learned from the survey is not surprising. Those 
individuals at sites having classified terminals were almost 
universal in rejecting the idea of an unclassified DROLS, some 
of them emphatically. Some said that without being able to get 
classified information on-line, the whole point was lost and 
there would be no reason for DROLS to continue existing. The 
essential rationales for this attitude were: 

1. The cost of a classified terminal is low compared to 
the value of the information that can be received in 


a timely manner from it. The researcher's time is a 
far more precious quantity. 

2. Many times quick response tasks are requested of re¬ 
searchers. Waiting for one or two weeks for informa¬ 
tion at the beginning of such a task would jeopardize 
their ability to meet the deadline. 

3. A number of researchers use an interactive mode, look¬ 
ing over the shoulder of the terminal operator while 

a search is being done. Thus, they are able to guide 
the search more efficiently and ensure that it is 
structured appropriately. 

On the other hand, the possessors of unclassified termi¬ 
nals were generally happy with what they had and were pleased 
to learn that their terminals would be capable of providing 
even more information to them in the future. They obviously 
did not object to DROLS being unclassified. In fact, due to 
the nature of their work, many users are not able to use clas¬ 
sified systems. The Army Library illustrates this point. It 
has a classified and an unclassified terminal, both of which 
are in substantial use. The classified terminal is used by 
members of the military staffs at the Pentagon for project and 
program information. The unclassified terminal is used to answer 
requests for information under the Freedom of Information Act, 
press and media requests for information, and thesis information 
for members of the Armed Forces. Since none of the latter can 
contain classified information, the unclassified terminal 
provides a desirable automatic screen that assures them that 
they can use freely whatever information they receive. 

D. COMMENTS ON OTHER FACTORS IN THE SURVEY 

Of our twenty-eight respondents, fifteen, or more than half, 
were librarians or information resource persons, most of whom 
actually used the terminals. The reason for the preponderance 


of persons of this type was that we had at least one from each 
organization who was the initial contact. In addition, some¬ 
times the contact was timid about volunteering information un¬ 
less his/her supervisor was present. In several cases both 
individuals were interviewed simultaneously, but even where 
separate interviews were conducted the response to any partic¬ 
ular question was the same. This seemed to be true at most 
organizations. That is, where three or four individuals were 
interviewed at an organization, they all generally felt the 
same about DROLS and its need for classification. Hence, it is 
not certain whether the library or resource people were simply 
reflecting the views of the researchers they served or whether 
they had formed their opinions independently. At one site, we 
believe that permission to respond to the survey had to be 
granted by an officer of the institution, and that he had given 
instructions about how to respond to the questions about clas¬ 
sification. All four respondents (two librarians) at this site 
were strongly opposed to an unclassified DROLS. 

This illustrates one possible weakness of our method of 
polling where the organizational hierarchy has the opportunity 
to review the questions and to set out the organizational point 
of view before the interviews are conducted. On the other hand, 
the views of management are given an opportunity to be repre¬ 
sented . 

This clearly did not happen at some other places. One 
site provided examples where opposite opinions were found in 
the same organization. One individual, a researcher, insisted 
that he needed information quickly in order to do his work. He 
declared also that if he could not get information right away 
there was a strong chance that he would no longer be interested 
in it. At the same site, a manager said that although he needs 
classified information for his work, he could be patient and 
wait for a DTIC search since he was never in that much of a 


hurry for the results. In fact, he much prefers the format of 
the DTIC search bibliography, a neat, stapled hard copy listing 
rather than the aluminized paper roll that he receives from 
the local terminal. 


Nearly every organization we visited had one or more ^ 

unique problems, thus making it more difficult and risky to 
generalize from this small sample to the entire possible 
spectrum of users. For example, one DoD organization, which 
one would think should have a classified terminal, has only • ^ 

an unclassified one, although they have been trying to obtain 
a classified one for some time. The problem seems to be some¬ 
thing as trivial as who is to pay for the installation of an 
electrical wall socket in the vault space designated for the 9 

terminal. In another unique situation, an agency has what 
seems to be a ridiculously limited budget for acquisition of 
reports, so it insists on having as complete information as 
possible on each report before‘it orders, since it must be sure • 

to order only reports that are needed. Despite these special 
circumstances we believe that the results of the survey identify 
the major needs felt by most DROLS users. In fact, if any 
group is unrepresented in this survey it is the non-users of # 

DROLS whose opinions were not sought; however, we did have some 
informal input from other contacts outside of the survey that 
indicated they were insufficiently aware of DROLS capabilities 
to make active use of the system. Nevertheless, they felt they • 

were getting good service from DTIC when they requested a DTIC 
search by mail or telephone. 


E. RESULTS (See Exhibits IV-2 and IV-3) 


To an overwhelming extent (21 out of 27), DROLS users in¬ 
dicated their preference for a classified DROLS over an unclas¬ 
sified system. The usual rationale for this was that their work 
required classified information and they had to have it reason¬ 
ably fast so that they could complete their tasks in the 
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EXHIBIT IV-2 


WHAT WE FOUND 


• 

Answers to question 14 

,,, WOULD YOU 

BE SATISFIED 

WITH AN 




UNCLASSIFIED DROLS? 





Unclassified 

Classified 



• 


Terminal 

Terminal 

Total 



Strong Yes 

2 

1 


6 


Moderate Yes 

1 

2 

3 f 


• 

Indifferent 

- 

1 

l 



Moderate No 

1 

6 

7 l 

21 


Strong No 

1 (Anomaly) 

13 

14 1 


• 

Rationales: Classifii£ 

,,, NEED CLASSIFIED INFORMATION, NEED 

RAPID 



RESPONSE, 

Cost is not 

A FACTOR 



Unclassified ,,, need unclassified information uncon¬ 
taminated BY CLASSIFIED FOR FOIA 
REPLIES, PRESS AND ACADEMIC SEARCHES 
(I,E., THESES AND PAPERS), No NEED 
FOR CLASSIFIED INFORMATION-CAN WAIT 
FOR DTIC SEARCH, ONE INDIVIDUAL PREFERS 
FORMAT OF DTIC SEARCH, 
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available time. This attitude was firm despite our suggestion 
that an unclassified DROLS would be much better in the future 
than it is today. But the users seemed to believe that they 
would be missing some citation that might be important if they 
could not get classified information on-line. Their point of 
view was that their time and the importance of the work they 
were doing justified any differential in costs between a clas¬ 
sified and an unclassified terminal. Only one respondent in 
answer to question 15 indicated that the cost of a classified 
terminal is too high. No other response indicated that anyone 
thought that cost was a factor in this question (e.g., see 
answers to question 5). Moreover, all of those we talked to 
already had sunk costs in their setup. Removing the classified 
system would waste most of the sunk costs. 

Although we must recognize the contrary views of our 
respondents, there are some strong arguments for going to an 
unclassified DROLS. For example, although the users want 
assurance that they are getting complete bibliographies, a 
number of factors besides classification limit the complete¬ 
ness of information provided by a classified search. We 
shall discuss these here: 

• Classification : By regulation DTIC has no Top 
Secret information in its files. Neither 

can it have cryptographic and communications 
security or communications and electronic 
intelligence. Moreover, the searcher and his 
organization must have a need-to-know in the 
particular areas in which he is searching. 

• Limited Distribution : Roughly 50 percent of the 
DTIC Technical Report files are limited in some 
way other than classification. Limits are placed 
by the originating agency of the study. They may 
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permit distribution of the entire summary 
to DoD, Government, or anyone. In addition, 
other limitations may be placed on contractors 
depending on their fields of interest. A dis¬ 
tribution limitation in DROLS denies only the 
abstract but not the complete citation of a 
limited document as it would appear in TAB. 

If a Technical Report is unclassified and 
unlimited, it is provided to NTIS, which then 
makes it available for public sale. This 
happens in the case of about 50 percent of 
all Technical Reports. NTIS leases the bib¬ 
liographic tapes of this information to one 
or more of the commercial data bases. 

Independent Research and Development File : 

Since the IR&D files contain information that 
is regarded as proprietary, no contractors, 
not even Federal Contract Research Centers 
(FCRC's), are permitted to access these files 
although they are available to DoD users. 

Data Not in Files : Despite clear, unambiguous 
regulations and directives regarding the submis¬ 
sion of reports and work unit data to DTIC (see 
Part A, Chapter II of this report) estimates re¬ 
veal that about one-half of the information that 
should be submitted to DTIC is not (Refs. 7, 8, 
9). We have little evidence of the reasons for 
this, but we infer that there are two major ones 




(1) the responsible individuals do not consider 
the submission of reports to DTIC as a serious 
responsibility, perhaps because there are no 
sanctions in the regulations or directives if 
they are not obeyed, and (2) there are some 
organizations or individuals in the Services who 
do not want to make public (even to the limited 
DoD public) what they have been doing. This 
suggests that DTIC should undertake some atti- 
tudinal research to determine how it might 
best obtain cooperation from those who are 
withholding information. 

• Purging of Files : To conserve on-line storage 
space, DTIC removes all records (except AD num¬ 
ber) in the interactive retrieval files for 
documents that are more than ten years old. 

Thus, a subject search on-line that might turn 
up older documents is incomplete and must be 
continued off-line by examining hard-copy docu¬ 
ments. The alternative is to order an off-line 
printout including a "delayed search," which pro¬ 
vides the full records for all the cited documents. 

Another argument for making the system unclassified is 
that it would be possible to interface with the various informa¬ 
tion and computer resource sharing networks (e.g., ARPANET) that 
can handle only unclassified material. This could allow greater 
use and flexibility of the system in acquiring and manipulating 
STI. Furthermore, an unclassified system could be controlled 
and managed by means of existing software packages of the type 
now used by the commercial data bases. This would overcome some 
of the more common criticisms of DROLS that we received from the 
users (see next paragraph (F) of this section). 



We conclude that a searcher in DTIC's files is usually 
going to receive only a large fraction (perhaps no more than 
half) of all the information that he should and would like 
to have no matter how efficient and skilled a searcher he is. 
This is true whether he has a classified or unclassified termi¬ 
nal. Using a classified terminal may result in receiving at 
most a few more citations than in the average search over an 
unclassified terminal, but it cannot ensure completeness until 
the other information flow inhibitors are overcome. 

So we conclude that a classified system may provide more 
information than an unclassified one, although at this point 
we do not have the data that tells how much more complete or 
useful for research and development such a system would be. 

Despite these positive reasons for converting DROLS to an 
unclassified system there are some strong reasons for not taking 
this action. First of all, most classified users are strongly 
opposed, because they feel that they would be denied significant 
information if they could not access classified files. Statis¬ 
tical arguments (about the proportion of classified information 
in the files) do not sway this point of view. The counter¬ 
argument is made by users that some areas of technology are 
nearly all classified and an unclassified search in such an 
area might result in few if any of the possible finds. They 
claim that it does not matter that there are many stronger inhib 
itors than security classification (see above), to the flow of 
information, the fact is that they have the need (and right) to 
obtain such information through a real-time system such as DROLS 

It is difficult to argue with this point of view: these 
users have in fact paid their dues by installing the proper 
facilities for receiving classified information and by having 
secure environments for the use and storage of such informa¬ 
tion. They see no reason why it should be denied to them by 
what might appear to be an arbitrary decision by DTIC. 


F. SOME OTHER COMMENTS ON DROLS 


All users were complimentary about DROLS and indicated 
that they were pleased with the service. Nevertheless, many 
of them had comments on how they would like to see the service 
improved. Nearly all of these comments came from the libra¬ 
rians or information resource persons who actually used the 
terminals. Although these comments are not new to DTIC, we 
believe that it would be of interest here to indicate to 
DTIC what some of the concerns of the users are in the context 
of this study. 

The gist of most of the comments we received was that 
they would like to have DROLS operate in the same way that 
Lockheed’s DIALOGUE and other commercial on-line services now 
operate. For example: 

• DTIC should provide a full Boolean search 
strategy. The partial Boolean now available 

is too limiting for sophisticated and efficient 
searching. 

• An on-line thesaurus would be helpful. 

• Also helpful would be the ability to narrow 
a search without starting over again. 

• A higher response speed on page shifting would 
also be desirable. 

• There is no convenient way of tracking from 
work units to the technical reports that 
resulted and vice versa. 

• Many documents such as proceedings of symposia 
and meetings and classified journals have indi¬ 
vidual articles in them that are not reachable 
as citations in DROLS unless they have been 
entered in the files separately. This results 


in loss of availability of many possibly valuable 
citations. The suggestion is to provide indivi¬ 
dual cataloging for the separate papers included 
in such documents. 


• The terminology in DRIT is years behind the 
current terms in such fields as electronic 
£ materials. This inhibits the ability of 

searchers in advanced fields to find citations. 
What the librarian would like is a more flexi¬ 
ble system that would enable the users to 
q search on their own terms which presumably 

represent the most advanced thinking in each 
field rather than being restricted to a dic¬ 
tionary of obsolete terms. Limited text 
• searching capability has been introduced in 

DROLS recently, but it does not fully meet 
this need. 


• • Some DROLS users also put program planning, 

work unit and technical report data into the 
system. It would be helpful to such users to 
have an on-line format generator. 

• 

• For documents older than ten years only an AD 
number is available on-line. Since ten years 
is an arbitrary length of time to decree that 
• work in technology is obsolete it would be 

desirable to provide some way of getting more 
information about such documents on-line. 

® • Many respondents are concerned about the time 

consumed by mailed reports and bibliographies 
getting to their, requestors even though they 
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are in the local DTIC area. We heard of delays 
approaching two weeks, although ten days is 
more frequent and six days seems to be a mini¬ 
mum. It is difficult to apportion these times 
among DTIC, the U.S. Postal Service and internal 
processing and distribution at the requestor's 
organization. DTIC claims to get reports and 
requested bibliographies into the mail in 4-6 
days. This could be an upper limit to delivery 
time if DTIC used a local courier service to 
deliver documents to Washington area users on 
the day they were ready. 


EXHIBIT IV-3 


SURVEY SCORE BOARD 



1. 

Please identify yourself: 

N 

% 

• 








a. 

Information Specialist 

1 5 

54 



b. 

Researcher 

8 

29 



c. 

Research Manager 

_5 

18 

• 



TOTALS 

28 

100 


2. 

Estimate your use of DROLS over the 





last 

year. 





a. 

Daily 

11 

39 

• 


b. 

Weekly 

5% (1) 

20 



c. 

Monthly 

10 

36 



d. 

Yearly 

l!s (1) 

5 



e. 

Never 

0 

0 

• 



(See Note 1) TOTALS 

28 

100 


3. 

What 

proportion of your collection 

0 - 

9 0% 



of documents is classified? 



• 

4. 

Have 

1 you used commercial on-line 





bibliographic services? 





a. 

Yes 

16 

57 



b. 

No 

7 

25 

• 


c. 

No Response 

5 

18 




TOTALS 

28 

100 


5. 

Why 

do you not have a classified terminal 

? 


£ 


a. 

No need for classified information 

1 

17 



b. 

Little need for classified 






information 

3 

50 



c. 

Cost 

0 

0 



d. 

Terminal procurement difficulties 

0 

0 

• 


e. 

Secure facility difficulties 

4 

67 



f. 

Information otherwise available 

2 

33 



g • 

Other 

L 

17 




TOTALS 

10(6) (3) 

184^ 


31 







% 


10 


12 


If only unclassified service were 
provided what would you do about 
missing information? 


expense of a classified terminal? 



a. Need classified information 

1 9 

83 

b. Response speed 

14 

61 

c. Cost not a problem 

3 

13 

d. Terminal provisions not a problem 

0 

0 

e. Other 

0 

0 

TOTALS 

36(23)^ 

1 57 

Would you consider changing to an 
unclassified terminal? 



a. Yes 

3 

13 

b. No 

17 

74 

c. No Response 

3 

13 

TOTALS 

23 

100 

How was the capital expense of your 
terminal funded? 



a. Library 

8 

29 

b. Organization 

7 

25 

c. Project 

2 

7 

d. Other or no response 

14 

50 

TOTALS 

32(28)^ 

111 

Who funds the operational expense of 
your terminal? 



a. Library 

7 

25 

b. Organization 

7 

25 

c. Project 

2 

7 

d. Other or no response 

14 

50 

TOTALS 

30(28) (3) 

107 


‘(2) 


‘( 2 ) 


'( 2 ) 


32 










a. 

Nothing 


5 

22 

b. 

Order classified 

entries 

11 

48 

c. 

Order classified 

reports 

1 3 

57 

d. 

No response 


3 

1 3 


TOTALS 

32(23) (3) 

112 


14. Would you be satisfied with an 

unclassified DROLS? (See summary 
of this question in Exhibit IV-2) 


15 . 


Why? 


a . 

No strong need for classified 
bibliography 

3 

11 

b. 

Can function adequately without 

4 

14 

c . 

Cost of classified terminal too 
high 

1 

4 

d. 

Can use DTIC searches 

5 

18 

e . 

Difficult to provide for classified 
terminal 

0 

0 

f . 

Strong need for classified 
information 

1 5 

54 

g. 

Classified search is faster 

16 

57 

h . 

Search more fruitful in-house 

1 4 

(28) v ; 208 


(See Note 2) TOTALS 

cn 

00 


NOTES: (1) One-half is used to designate the situation where 

a user was uncertain between two intervals of the 
time scale. Answer "e" is as expected since the 
respondents were selected as users. 

(2) Totals greater than 100% indicate that one or more 
respondents gave multiple answers to the question. 


(3) Number in ( ) is actual number of respondents to 
question. 







V. DISCUSSION 


A. GENERAL 

As we have noted earlier, DTIC is considering changes in 
its on-line system that would extend its responsiveness through 
making it possible to use value-added communications networks, 
and that would make possible the use of commercial software for 
the operating system. The barrier to potential improvements is 
the problem of security classification. If DROLS could be un¬ 
classified, then more operating economies and improvements could 
be made, but if DROLS must remain classified then the system 
may have been improved about as much as it can be. 

B. DROLS FUNCTIONS 

DROLS presently performs two functions. One of these is 
to provide bibliographic services to individuals and organiza¬ 
tions engaged in research and development. The second is to 
provide management information to program managers and manage¬ 
ment offices. Even in this second function, DROLS is severely 
limited in that it does not provide precise management informa¬ 
tion. The Work Unit files and the Program Planning files, for 
example, tell who is doing what and where it is being done. 

They offer only order-of-magnitude cost information; for more 
accurate information it is essential to contact the agency 
that is actually doing or sponsoring the work. The Program 
Planning and Work Unit files are no better than a bibliography; 
they tell where to find the real information that one might be 
looking for. 


It is clearly desirable that the DTIC system provide more 
direct on-line access to and manipulation of actual R&D program 
data rather than bibliographic-type reference. The problem 
here is that security restrictions made such changes very diffi¬ 
cult if not impossible. Due to these restrictions the system 
is inflexible, which means that data bases cannot be added with¬ 
out inordinate difficulty to ensure that they are properly pro¬ 
tected. It is not possible to program on-line, nor is it possible 
for DTIC to use commercial program packages such as those used by 
Lockheed and SDC to identify and verify users and to operate the 
system. Not being able to use such packages presents a major 
disadvantage to DTIC, which is forced to do all its programming 
internally, and which is saddled with an inflexible system that 
is difficult to improve or expand. 

The task statement mentions a dichotomy that exists 
between the DTIC mission to disseminate information and the 
requirements for security of the classified information that 
is included in the files. The authors assert that no real 
dichotomy exists. Rather, the primary function of DTIC is to 
store and disseminate information to all those who need it. 
Security regulations only limit dissemination by defining who 
needs the information and by prescribing the means to be used 
to prevent others from obtaining it. 

C. ALTERNATIVES FOR DTIC 

We discuss four courses of action for DTIC with respect to 
dealing with the on-line situation discussed above. 

1. Continue DROLS as the combination classified-unclas¬ 
sified system that it currently is. Recent modifica¬ 
tions of this system enable DTIC to provide uncleared 
users with almost the same service that cleared users 
receive. .Further planned changes will make this dif¬ 
ference even smaller in the future. 


2a. Change DROLS to a completely unclassified system. 

This will simplify making the kinds of improvements 
that many users would like to see, by making it pos¬ 
sible for DTIC to purchase software packages that 
security requirements will not permit and will reduce 
operating costs for both DTIC and its users. But this 
also reduces the ways available to cleared users to 
obtain classified information. 

2b. Completely unclassify the external portion of DROLS 
but retain the internal classification so that clas¬ 
sified searches can be done in-house. This would re¬ 
duce costs to users while increasing their inconven¬ 
ience. Classified bibliographies could still be 
obtained by ordering a DTIC search. This mode would 
save DoD and DTIC some money because the DTIC fac¬ 
ilities in Los Angeles and Boston would no longer be 
needed. Also, there'would be savings in communications 
costs and in the maintenance of crypto equipment that 
would accrue ultimately to the DoD. However, the 
savings and convenience to DTIC will be limited be¬ 
cause it must retain computer security for its in- 
house terminals. 

3. An intermediate option between these alternatives is 
to create two essentially separate systems, one clas¬ 
sified, the other unclassified, using two computers 
and two separate and different modes of communication. 
Then it would be possible for DTIC to improve the 
unclassified files in any way that it wanted without 
being concerned about security. This might also 
simplify the problem of making improvements to the 
classified system. 

Alternative 3 presents some problems of its own. For 
example, the cleared user, in order to do a complete search. 


37 


would have to search two data bases or at best initiate the 
search differently. Furthermore, it is not clear how the in¬ 
formation should be split between the bases. One way of di¬ 
viding up the files is by fields, placing all classified fields 
in the classified computer and everything else in the unclas¬ 
sified computer file. With such a scheme the uncleared user 
would be served about as well as today, while the cleared user 
would have to access two files. This also creates an input 
problem for DTIC since the division of the files for this pur¬ 
pose would have to be accomplished carefully to ensure that 
nothing classified was stored in the unclassified computer. 

Since this alternative creates problems for both DTIC and its 
classified users, we believe it to be the least desirable al¬ 
ternative, especially considering how effective the unclassified 
DROLS has become. Moreover, the present system handles reliably 
differences in classification from Unclassified to Secret as 
well as need to know. On top of this it also establishes effec¬ 
tive control over limited distribution information 

Also, we think it inadvisable to adopt alternative 2a or 
2b at this time. The reason for this is politic and psychological 
rather than logical or economic. Our survey of users (see 
Chapter IV) shows that they would object strenuously to changing 
the system in this way, even though they are not likely to 
notice any essential difference in the way that it operates. 

But, rightly or wrongly, they do feel that they would be denied 
some vital information if they could not obtain it through their 
terminals, and they would resent the delay involved in trying 
to obtain references by other means. One aspect of this is 
that the cleared users seem to consider themselves as belonging 
to some kind of exclusive club, to which they have, in some 
sense, paid their dues by having a classified terminal and fa¬ 
cility and they do not want to lose any of the benefits of mem¬ 
bership, even though such benefits may be largely illusory. 
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# 




# 




There may be ways to overcome this kind of objection to 
an unclassified system. For example, even though a cleared 
terminal could not receive classified information in an unclass¬ 
ified system, it would be given the unclassified fields as 
unclassified terminals now receive. If any classified inform¬ 
ation is denied to a formerly classified terminal because the 
system has been made unclassified, then the document itself 
could be automatically sent to the requestor (perhaps without 
charge) thereby demonstrating a "membership benefit." 

D. ALTERNATIVES FOR USERS 

Becoming unclassified does reduce the utility of DROLS to 
its users who now have classified terminals. But, in fact, the 
unclassified DROLS has been improved so much that the classified 
user will receive little more information than will the unclas¬ 
sified user. Moreover, making the DROLS unclassified does not 
deny the user the classified information that he needs and would 
like to have. To get such information if he has only an unclas¬ 
sified terminal, the classified user must request DTIC to do a 
batch search and to mail the results to him. Thus, there is a 
delay of perhaps six to fourteen days while the user waits for 
the results of his search. (We presume here that DTIC retains 
the ability to do such searches in-house.) There is still an¬ 
other alternative available to the user who may be in a hurry. 

He can simply order any classified documents that his search 
reveals without waiting for the complete citation. In this case 
he may receive some documents that may not be useful to him, 
but he can receive those documents as quickly as though he might 
have seen the complete citation. In addition, users who are 
knowledgeable about the technical fields they are working in 
know other sources of the information they are seeking and do 
not need to rely on DTIC for all of their information. 
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So, in effect, what we are dealing with primarily is some 
extra time to obtain partial bibliographical information. In 
fact, this extra time need not be spent if the user is willing 
to order reports from less than complete citations, and take 
the chance that some of those reports that he orders in this 
way may not be useful. If we try to examine the value of time, 
here is what we run into. 

E. VALUE OF TIME 

The question of the value of time and information is tricky. 
It might seem possible to assess the cost of a delay in obtaining 
information by adding up the cost of additional researchers' 
time spent in waiting. But this assumes that the researcher 
is nonproductive while waiting, when in fact he or she could be 
working on another project or studying unclassified literature 
until the classified material arrives. Another way of looking 
at the value of delayed information would be to examine the cost 
to an organization of not winning a contract because a proposal 
was late or incomplete due to missing information. But this 
presumes that the quality of the proposal relative to other 
proposals remains the same except for the contribution of the 
new information. Still another way of assessing the value of 
delayed information would be to determine the cost of duplication 
of an R&D effort because information was not available in a 
timely way. 

With any of these methods of assessing the value of in¬ 
formation, there is a presumption that the researcher and re¬ 
search manager are not knowledgeable about what is happening 
in their own fields and that they do not have the resources or 
resourcefulness to use other means of getting timely information. 
Included among these other means are personal contact with in¬ 
dividuals and organizations in their own areas of expertise. 

Still another way of getting information can be inferred from 


the fact that DROLS provides only bibliographic information. 

If "real" data and facts are needed, the only way that DROLS can 
help is to tell the user where to find them, including knowledge¬ 
able individuals, if requested. Citations to classified reports 
will be given to all users, classified and unclassified. However, 
the classified citation is likely to be somewhat more complete 
than the unclassified. Hence, the user receiving unclassified 
information will have less information to decide whether or not 
the report will be useful. If he is uncertain and if it is im¬ 
portant enough he can order the report and receive it after the 
same delay that the cleared user might undergo. 

Reference 10 reports on a client survey conducted by the 
NASA Industrial Application Center at the University of Southern 
California. Telephone interviews were conducted on a random 
sample of clients. Of the total 159 clients interviewed, more 
than 53 percent identified dollar benefits of on-line searching. 
Furthermore, a direct relationship was shown between client 
dollars invested and benefits derived from the research. The 
average ratio of benefits to investment was 2.9* We note that 
this ratio is a result of the perception of benefits as deter¬ 
mined by user and that in some cases the ratio was considerably 
higher (about 10-20). While we would question this method of 
gathering data on this subject, there seems to be no question 
in most client's minds that they benefited significantly from 
the information service. 

Thus, we conclude that determination of the value of infor¬ 
mation would be extremely difficult with results that are not 
likely to be credible because they involve unacceptable assump¬ 
tions. Moreover, the denial of information by an unclassified 
DROLS is a denial only of partial bibliographic information and 
is in no sense a denial of the "real" information that the user 
would like to have. It only impinges on the decision as to 
whether or not to order a report or reports that the biblio¬ 
graphic record indicates might be useful. Considering the 
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general incompleteness of the bibliographic record (see Para¬ 
graph IV-E), we believe that the classified DROLS offers very- 
little more to the user than would a completely unclassified 
DROLS. 


VI. FINDINGS AND CONCLUSIONS 


* 


» 


A. FINDINGS 

1. DROLS Service 

DROLS serves a growing body of users from about 100 remote 
terminal sites. The system provides both classified and unclas¬ 
sified bibliographic and other services to a variety of users, 
most of them DoD, or affiliated with DOD. The service is heav¬ 
ily used by a few users and only moderately used by most, but 
the instances of dissatisfaction that we found were related not 
to the basic service, but to the fact that commercial on-line 
services (unclassified) have surpassed DROLS in their ability 
to provide ease of access to and use of a data base. Most such 
comments were variations of "Why can't DROLS do it like _?" 

2. Limitations on Effectiveness 

Searching for information in the DROLS files is severely 
hampered by the fact that (1) the files are incomplete, thus 
reducing citations available on-line, (2) information is not 
in the file because it has a higher classification than DTIC 
is authorized to receive or because it is not submitted to DTIC 
despite the fact that regulations require such submission. The 
net result of these omissions is that a searcher can receive no 
more than about one-half of the references that he would like 
and should receive for completeness. Since only about seven 
percent of the citations in the TR file are classified, the 
effect of classification on denied information is likely to be 
a small component in these proportions. 
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3. Classified versus Unclassified 


A survey of a sample of users indicated that those who now 
have classified service would be very reluctant to trade that 
off for a greatly improved unclassified service that would pro¬ 
vide nearly all the bibliographic information that the classified ^ 

system includes. 

4. Costs (see Appendix B) 


For a representative site, the one-time costs of installing a 
classified terminal are about $11,000, while an unclassified ter¬ 
minal could be obtained for an initial one-time installation cost 
of less than $100. We assume both kinds of terminals are leased. 
Based on actual IDA experience, monthly operating costs for the 
classified terminal would be about $1,000 while the same for an 
unclassified terminal might be $145 plus telephone line costs. 
Telephone line costs can be a large variable depending on the 
location of the user’s facility and how much connect time is used. 
It may be more economical for some non-local users to have a ded¬ 
icated telephone line even though they have an unclassified ter¬ 
minal and could use a dial-up access mode. Since we did not 
interview people who have no terminal, we have no notion of 
whether such costs deter organizations or individuals from ac¬ 
quiring a terminal or whether the difference between the two 
would influence the selection of terminal classification. But 
among the terminal users whom we did interview, the universal at¬ 
titude was that the information and the convenience in obtaining 
classified or '’all” information are well worth whatever it costs; 
on the other hand, these individuals do not pay for these serv¬ 
ices. Their organizations, and ultimately the DoD, do. 

DTIC in-house costs may not be significantly reduced by 
going to an unclassified DROLS unless DTIC also gives up doing 
classified searches. If classified searches are provided, then 
the entire internal operating system must remain classified and 
security arrangements at the Cameron Station facility would not 
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be significantly changed by having an unclassified DHOLS. How¬ 
ever, savings could be realized by closing down the classified 
Boston and Los Angeles offices, which would no longer be needed. 

Another change, which we discussed earlier, would be to 
institute a dual system with separate classified and unclassified 
parts. Here again it is difficult to see much savings from such 
a system with the possible exception of using commercial software 
for changes in the unclassified system. But since the classified 
services would continue to be offered, such cost savings could 
be only minimal. 

DTIC might also make DROLS completely unclassified, 
internally as well as externally. Then all security costs assoc¬ 
iated with DROLS could be eliminated, but the savings would be 
limited because DTIC itself must remain a classified facility 
since it must store and supply classified documents for all 
DoD agencies. Other savings would accrue to both DTIC and the 
DoD from eliminating the need for maintenance of cryptographic 
equipment and from users optimizing communications charges by 
using dedicated lines or dial-up, depending on their volume of 
business and distance from Cameron Station. Many local users, 
for example, who now pay for dedicated lines, would have virtu¬ 
ally no communications costs if DROLS became unclassified. 

5. Possible Improvements 

DTIC clearly desires to improve its services to its clients. 
The clients would welcome better service and some have suggested 
what they would like. The most common suggestion was that DTIC 
should modify its software to permit searching in a similar way 
to that permitted by the commercial on-line services, such as 
Lockheed or System Development Corporation (SDC). This problem 
appears to stem from DTIC using "home-grown" software rather 
than commercially available programs. Unfortunately, security 
requirements prevent the use of commercial software. These 
improvements include complete Boolean search strategy, on-line 
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indexing and retrieval terms, and elimination of look-up codes 
for organizations and subject areas. We are aware that these 
suggestions have been made before, but we feel that it may be 
useful to repeat them in the context of this report. A more 
complete list of user suggestions can be found in Chapter IV. 

B. CONCLUSIONS 

• There is no real dichotomy, as mentioned in the 
task statement for this study, between the services 
that DTIC is designed to offer and security 
requirements. Security requirements only state 
limits as to who is to receive certain kinds of 
sensitive information, whereas the mission of 

DTIC is to disseminate information as widely as 
possible, within security limits. In fact, 
security limitations are only one of the lesser 
limiters of DTIC information disseminated with 
the aid of DROLS. 

• The most important limiting factor on DROLS 
effectiveness is that many documents are not 
submitted to DTIC and hence not represented 
in the files. About 50 percent of all docu¬ 
ments that should be submitted are missing 
despite regulations and directives that require 
such submission. Still another important limit¬ 
ing factor is distribution limitations that 

are frequently used to limit the flow of infor¬ 
mation . 

• The unclassified service that will be offered 
on DROLS in the next few months will be vir¬ 
tually as complete as the classified service. 

• The order of overall desirability of alterna¬ 
tives for the future of DROLS is the following: 

46 


* 


1 . 


m 



Retain the present classified and unclassified 
system. 

2. Make DROLS service to external terminals 
completely unclassified, but retain in- 
house classified service. 

3. Make DROLS completely unclassified, including 
in-house service. 

4. Convert the present DROLS into two separate 
systems, one classified, the other unclassified 

• DTIC users have suggested ways to improve DROLS 
that seem desirable. Most common among these is 
the suggestion to design protocols for addressing 
and using the system similar to the commercial 
on-line data bases. 

• Finding ways to increase the completeness of its 
files and creating a more user-oriented on-line 
system would be a researchable and desirable 
activity for DTIC to engage in. 

• Users who currently have classified terminals 
will object strenuously to changing to an un¬ 
classified DROLS. To offset such objections, it 
is suggested that such users would be given 
certain privileges, that would not be given to 
unclassified users, such as automatic sending 

of any classified document cited either wholly 
or partially during a DROLS search by a classified 
user. 




47 


REFERENCES 


1. DTIC Digest No. 8l, Defense Technical Information Center, 
February 10, 1980. 

2. Scientific and Technical Communication , Committee on 
Scientific and Technical Communication, National Academy 
of Sciences, 1969 . 

3. Georges Anderla, Information in 1985 , Organization for 
Economic Cooperation and Development 1973. 

Jerome T. Maddock, et al., DDC 10-Year Requirements and 
Planning Study , Survey Results Report, Auerbach Associ¬ 
ates, Inc., AD-A022 304, March 14, 1976. 

5. Thomas J. McGeehen & Jerome T. Maddock, DDC 10-Year 
Requirements and Planning Study , Interagency Survey 
Report, Auerbach Associates, Inc., AD-A022 302, 

December 12, 1975 

6. Andrew A. Aines, A Critical Study of the Department of 
Defense Technical Information Program Management--Direction 
for Tomorrow , OUSDRE, R&AT, January 1978. 

7* Especially DDC—Users Look, at the DoD Information Transfer 
Process , Washington, D.C. Committee on Information Hang-ups, 
AD-A005 400, p. 21, January 1975. 

8. SBIE: Background of the Experiment , Panel Report to Special 
Library Association at a meeting in Kansas City, MO, Summer 


9« Institute for Defense Analyses, Shared Cataloguing Experi¬ 
ment Within the Defense Community , Proposal submitted by 
IDA Technical Information Services to DDC, December 1976. 

10. Rebecca J. Jensen et al., Costs and Benefits to Industry of 
Cn-Line Literature Searches , NASA Industrial Application 
Center, University of Southern California, Special Libraries, 
July 1980. 


49 


APPENDIX A 


SUMMARY OF RESEARCH PERFORMED 


* 




A-l 


APPENDIX A 






SUMMARY OF RESEARCH PERFORMED 

In December 1979, the Defense Technical Information Center 
(DTIC) requested the Institute for Defense Analyses (IDA) to 
undertake a study of its information retrieval system known as 
the Defense Research, Development, Test and Evaluation (RDT&E) 
On-Line System (DROLS). The original work statement involved 
five subtasks. Subsequently, after some work had been done on 
each of the subtasks, DTIC announced some changes in DROLS that 
had not been scheduled when the original work statement was 
drafted. Accordingly, the study task was redirected by mutual 
agreement to reflect the later status of DROLS and the work 
proceeded. The redirection changed three of the five original 
task statements, one was deleted and one was unchanged. In 
this summary we shall review the task statements, both original 
and redirected, and indicate briefly what was done about each 
and where the details of the task investigation can be found in 
this report. 

A. SUBTASK A 

This subtask requested IDA to consider principal objectives 
and priorities involved in the operation and design of DTIC’s 
on-line Scientific and Technological Information (STI) data base 
systems in support of the Defense RDT&E community. In response 
to this subtask we reviewed relevant documentation including 
regulations and directives and discussed the subject generally 
with a variety of DROLS users. We found a vast number of regu¬ 
lations and directives have been issued by the Department of 
Defense and all of the military services that relate to the 
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operation of DTIC and DROLS (refer to the References in Part A 
for a sample listing). In particular, descriptive material for 
all ongoing and completed RDT&E projects must be submitted to 
DTIC. The methods and responsibility for submitting this infor¬ 
mation are detailed in these regulations. There is no doubt 
about what should be done. Yet many military agencies ignore 
them and fail to submit the information that they are required 
to submit. The net result of this neglect is that the DTIC 
files are incomplete. The amount of information missing ranges 
up to 50 percent, according to some estimates. It would seem 
that with such a vast amount of information missing, DTIC would 
be severely crippled in the performance of its primary function 
of being the central depository and single access point for all 
Defense RDT&E information. The potential danger involved is that 
the flow of information is grievously impeded, resulting in ig¬ 
norance of important research findings, duplication of effort, 
possibly unbuilt or poorly performing systems and greater cost¬ 
liness of U.S. Defense RDT&E efforts. It is clear that the prob¬ 
lem cannot be handled by the present set of regulations. The 
present regulations are complete; but they are largely ignored. 
Unless some teeth can be put into the regulations and enforce¬ 
ment encouraged from the top down, the situation is likely to 
continue. Work on this subtask, which was not changed, is dis¬ 
cussed further in the first volume (Part A) of this report. 

B. SUBTASK B 

This subtask originally required IDA to review the current 
secure communications system used by DROLS. We also were re¬ 
quired to identify unclassified on-line services that could 
access unclassified data and determine the principal character¬ 
istics of communications services, including their costs. In 
response to this original subtask we developed a model repre¬ 
senting a computer-communications system. The model provides a 


method of calculating the various principal characteristics of 
the system, and is appended to Part A of this report. 

This subtask was redirected to require a survey of a sample 
of DTIC users to obtain their reactions to the proposition that 
DROLS become a completely unclassified system. We found an over¬ 
whelming majority of those DROLS users who had classified ter¬ 
minals insisting that they remain so, while the unclassified 
users were, by and large, satisfied with what they could get 
from DROLS. This survey is described and the results provided 
in detail in Chapter IV of this volume. 

C. SUBTASK C 

This subtask initially requested us to look at the question 
of using unclassified terminals to access a data base, part of 
which may be classified. 

Since DTIC has been doing exactly this for almost a decade 
with no evidence that its classified information has ever been 
compromised by an on-line user at an unclassified terminal, the 
question seems somewhat academic. Moreover, there are now com¬ 
mercially available a variety of "security kernels" that have 
the capability of handling multi-level classification schemes in¬ 
cluding such items as the need-to-know. However, these capabil¬ 
ities of commercially available software have not been verified 
and consequently this software is not yet approved for DoD pur¬ 
poses. Another problem with any such scheme is that the remote 
terminal must be in a properly secured area and access to the 
terminal itself must be given only to those who have appropriate 
clearances. If such control of the remote ends of the system 
cannot be achieved then the entire system could be compromised. 
However, there seems to be no real problem in maintaining the 
system security if the classified terminals are kept secure, 
even though both classified and unclassified users can access 
the files. 


Subsequent to the work reported above, subtask C was re¬ 
directed to require that we determine the need and utility of 
classified information as available from DROLS to representa¬ 
tive system users. 

To determine this need we relied on the survey mentioned 
under subtask B above. The general findings are that classified 
users need essentially complete information to do their work 
effectively, and they are not willing to have their search 
results truncated because of classification. See Chapter IV 
for more detail. 

D. SUBTASK D 

This subtask requested us to examine the possible impact 
of AUTODIN II on DROLS if it became the universal telecommuni¬ 
cations medium for all DOD installations. 

Before this task was redirected we had the time to give only 
some very preliminary thought to this question. The response 
time analysis that was done in answer to part of the original 
subtask B described above applies in part here (see Appendix D, 
Part A). However, this analysis was abandoned when the task was 
redirected since it needed more data for completion. 

The changed task requested us to provide some cost informa¬ 
tion for both classified and unclassified terminals. We could 
not find any significant difference in manpower costs due to 
security, but there are some differences in the one-time and 
operating costs of the two types of terminals. Costs are pre¬ 
sented in Appendix B. 

E. SUBTASK E 

This subtask requested us to examine the possible impact of 
office automation developments, including the paperless office 
and laboratory to determine how such developments may enhance 


the functions of DROLS. This subtask was deleted and no further 
report appears in this paper beyond the following paragraph. 

Until the task was redirected we managed to accomplish an 
initial gathering of information and only some very preliminary 
thinking about this question. There did not seem to be much 
applicability to DROLS at present, but teleconferencing, full 
page image transmission, and systems based on film or video 
disc, perhaps controlled by computer, may be promising applica¬ 
tions for information processing and transfer to the user. 
Generally, paperless operations might enhance DTIC internal op¬ 
erations much the same as they would enhance almost any organi¬ 
zation's internal operations. 
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THE COST OF HAVING A REMOTE TERMINAL 

We present here three sets of costs, two of which show the 
costs of acquiring and operating a classified terminal, and one 
of which shows the same for an unclassified terminal. The two 
classified terminals costs apply to different times ... 1969 
and 1980. In this approximately one decade, costs have not 
varied much despite the general inflation in the economy. In 
fact, operating costs have actually diminished. The primary 
reason for such cost behavior must be attributed to the great 
advances that have been made in the technology of electronic 
data processing in this period. 

The 1969 cost data were taken from a DDC report by 
Richard K. Bennertz that described the various phases involved 
in the development of the DROLS (Ref. B-l). The other two sets 
of costs are actual site costs that might apply to a user in 
the current time. These costs are IDA costs and reflect IDA's 
particular situation. IDA already had a vault and the only re¬ 
quirement was to find some work space for the terminal, crypto 
equipment and an operator. So site preparation costs were 
fairly nominal in such a case. Also, monthly operating costs 
are highly dependent on the user's proximity to DTIC, which af¬ 
fects primarily the costs of a dedicated telephone line, or if a 
dial-up mode is used it will affect user charges. For example, 
a local call for IDA would cost essentially nothing, but a call 
might cost a substantial amount from a more remote location 
such as the U.S. West Coast. On the other hand for an unclassi¬ 
fied terminal the one-time costs are quite nominal, assuming 
the terminal will be leased rather than purchased, although even 
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here, prices have been coming down rapidly so that the purchase 
of a terminal could hardly be considered a major expense for 
most DTIC users. 

There are some other costs involved in having a classified 
terminal. These include some share of the security overhead, 
the need for having crypto-cleared individuals available, and an 
operator. However, in no case could we find that these costs 
added anything significant that could be charged to the terminal. 
For example, the security force is not likely to be increased 
due to a few square feet of vault space dedicated to a terminal 
in any organization that has to maintain a secure environment 
anyway. Similarly, the individuals needed to operate the crypto 
equipment have to be on hand for other reasons and the amount 
of work the crypto requires involves a tiny fraction of their 
time. The operator’s job is the same whether or not the terminal 
is classified. The only distinguishable additional costs of a 
classified terminal are maintenance of the crypto equipment and 
the monthly costs of the dedicated telephone lines that are nec¬ 
essary for classified terminals. We note that it may be less 
costly for non-local unclassified terminals to use a dedicated 
line, depending on connect time and distance from DTIC. 

It can be concluded that the cost differences between clas¬ 
sified and unclassified terminals are noticeable. Whether they 
are significant or not depends on the user's needs and attitudes 
toward achieving the desired result of complete data. As we have 
noted in Chapter IV, this result is currently unattainable any¬ 
way, even with a classified terminal, primarily because all the 
information that should be in the DTIC files is not there. 

We note also that there are additional cost factors to DTIC 
(and to the DoD) in maintaining the classified DROLS. Among these 
are DTIC maintenance of crypto equipment, two remote offices (in 
Boston and Los Angeles for local classified users) and the non- 
optimum communications arrangements that ensue fromthe requirement 
for dedicated lines. 
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COSTS OF REMOTE TERMINALS 


1969 CLASSIFIED (Ref. 

D- 

1) 

IDA - 1980* CLASSIFIED 

One-Time Costs 



One Time Costs 


KG-Crypto Unit 

$ 

3750 (GFE) 

Crypto Aux. Unit 

$ 3000 

Junction Box 


300 

KG 13-Crypto Unit 

3448 

(GFE) 

Telephone Installation 

938 

Telephone Cable 




Ass 'y 

68 

Site Preparation 


3800 

Installation 

4000 

A1 lied Support 


2636 

Site Preparation 

750 

Installation Team 


1037 

Safe (Including 





Shipping) 

3050 

TOTAL 

$12461 

TOTAL 

$10868 

Monthly Operating Costs 


Monthly Operating 

Costs (1980) 

Terminal 


895 

Maintenance 

167 

Modem Circuit 


566 

Telephone Line 

181 




Modem Lease 

250 




Terminal & 

Peripherals Lease 

418 

TOTAL 


1461 

TOTAL 

$1016 


1980 UNCLASSIFIED SYSTEM 



One-Time Costs 

Monthly Operating 

Costs 

Installation $70 

Maintenance 

145 


Telephone (Time 



& Distance Dep.) 

0 


TOTAL 

$145 


*Although the IDA terminal was installed in 1979 as a classified 
terminal, we believe that these costs are reasonably close to 
actual 1980 costs. 
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SURVEY OF DROLS USERS 

As reported in Chapter V of this report, surveys were con¬ 
ducted at a total of eleven user sites, including IDA. In pre¬ 
paration for this survey, the survey questions were sent to the 
potential respondents in advance, along with some background in¬ 
formation and a cover letter signed by the DTIC Administrator. 
This Appendix presents all this material in its original form 
except for the footnote on page C-7, which has been added here 
to indicate a change that we learned about after the survey had 
been completed. 
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DTIC-I 


Ly t i • o — — * --J * O f « O ■■ > ■ - - ■ 


t\ L EXAND P l A, V 1 3G! I\i * A 22 31 ■-* 


1 August 1980 


Gentlemen: 

As an effort to reassess and improve our technical information services, • 

DTIC has contracted with the Institute for Defense Analyses (IDA) to help 
us evaluate possible ways to improve the overall effectiveness and 
efficiency of the Defense RDT&E On-Line System (DROLS). Your organization 
has been selected as part of a small sample of DROLS users from which to 
gather information about DROLS usage. Enclosed please find four copies of 
a questionnaire that IDA has prepared. We ask that you have three members ® 
of your organization’s staff examine them. The three persons should be 
selected as follows: 

a. A technical information services person familiar with DROLS. 

b. A technical project manager who supervises bench-level personnel. ® 

c. A bench-level technical person whom you know uses DROLS. 

In a few days, an IDA staff member will contact you to conduct interviews 
of the individuals you have selected. The interview will focus on the 
questions in the enclosed, which are presented here so that you will have a 
preview of the scope of the questions and be able to do some preliminary 
thinking about the answers. 

I would appreciate your cooperation in this matter and in assisting IDA to 
accomplish their assigned effort. 0 

Sincerely, 

HUBERT E. SAUTER * 

Administrator 
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QUESTIONNAIRE FOR DROLS USERS 


I. INTRODUCTION 

Under a contract with the Defense Technical Information 
Center (DTIC), the Institute for Defense Analyses (IDA) has 
been tasked with evaluating a possible change in the DTIC- 
operated Defense Research On-Line System (DROLS). Your faci¬ 
lity accesses bibliographic information from DTIC. We have 
elected to query you about attitudes toward the use of DTIC 
services because you are among the heavier users of DROLS. 

In the following paragraphs we shall describe several pro¬ 
grammed changes in DROLS and then we shall describe a possible 
further change. We should like to have your comments on this 
change, in accordance with a set of questions that follows the 
descriptions. 

II . PROGRAMMED CHANGES IN DROLS 

Three changes in DROLS are to be implemented this year 
(1980). These changes will affect only users who have unclas- 
sified terminals. The first change, implemented in July 1980 , 
permits the user of an unclassified terminal to access unclas¬ 
sified entries in the data file for technical reports.* That 
is, unclassified entries (the only classifiable fields in an 
entry are: titles, descriptors, identifiers, and abstracts) 

* 

While there are four files in DROLS, the examples of 
improved access are stated in terms of Technical Report 
file. 





will become available to an unclassified terminal. About 90% 
of the Technical Reports and their entries are unclassified. 

Of the approximately 10 % of the Technical Reports that are 
classified, 31 % have completely unclassified entries. The 
latter, except for some abstracts which have limited distribu¬ 
tion, will be fully accessible to an unclassified user. In 
other data banks, the proportion of classified citations (cita¬ 
tions are the equivalents of Technical Report entries) is 
even lower than that of the Technical Reports, so an even 
higher proportion of these data banks is available to unclassi¬ 
fied terminals.* If a classified report or entry (of which the 
existence has been determined by searching the technical Ab¬ 
stract Bulletin or DROLS) is wanted by a searcher, it can be 
requested on-line or by mail and it will be mailed by DTIC in 
an average of 3-^ days.** 

The second change, to be implemented in October 1980, will 
be the initiation of a dial-up capability for any unclassified 
terminal user registered with DTIC. With this system the user 
can access DTIC with almost any type of terminal and telephone 
line or TYMNET. This change will reduce substantially the cost 
of accessing DROLS. 

The third change, to be implemented in about November 1980, 
attempts to overcome a major deficiency in the first change 
noted above. All unclassified fields from a classified entry 
will be accessible to an unclassified DROLS user; that is, 
unclassified titles, descriptors, identifiers, and abstracts. 


Access, similar to that to be made in July 1980 to the 
Technical Report Pile, is already available to the Work 
Unit and Program Planning Files. The IR&D file is acces¬ 
sible only to government personnel since it contains pro¬ 
prietary data. 

We made no estimate for the time required for delivery of 
registered U.S. mail and for the internal distribution of 
mail at the receiving organization. 



Only certain limited distribution abstracts may not be avail¬ 
able to all subscribers. For this planned change the classi¬ 
fied user and terminal would perceive no difference from the 
current system. 

Unclassified terminals would be capable of receiving rela¬ 
tively complete bibliographic information. We estimate that 
about 7% of the entries for all technical reports ( 69 % of the 
entries for classified technical reports) will have from one 
to four fields (title, descriptors, identifiers, or abstracts) 
that will not be accessible to unclassified terminals. Never¬ 
theless, a searcher at an unclassified terminal, in response 
to a subject search, would be provided at least an AD number, 
organization author, personal author, and report date. There¬ 
fore, such a searcher will know that a classified report exists 
that matches the inquiry. An exception to this exists: No 
find will be reported on-line if the searcher performs a sub¬ 
ject search and the matching term in an entry is classified or 
is in a classified field.* If an unclassified terminal user 
wants a report or classified entry found by a search, it must 
be ordered for delivery by mail. If a comprehensive search 
including classified entries is wanted, an off-line biblio¬ 
graphy must be ordered. Again, we emphasize that the informa¬ 
tion missing from the on-line system, in any case, is only 
bibliographic information. Reports or other documents must 
still be requested and delivered by mail. 

Distribution limitations would still apply. Therefore, 
except for classification restrictions, almost all abstracts 
would be available to DoD users, fewer to other U.S. government 
users, and still fewer to DoD contractors, as is the case today. 

* 

We have been informed by DTIC that this exception will no 
longer exist due to a modification planned to the system 
later this year. Unfortunately, we received this informa¬ 
tion after the survey was completed. 



Hereafter, we shall refer to the DROLS with the above 
three changes as the "Baseline" System. 

III. POSSIBLE CHANGE IN DROLS 

The classified on-line component of the Baseline System 
would be discontinued. 

The unclassified on-line component of the Baseline System 
would continue. Those title, identifier, or abstract fields 
that are classified would not be accessible. The descriptor 
field would be improved to contain all the posting terms (from 
the DRIT) that are applicable to the report so that subject 
searches would be more successful than searches from an unclas¬ 
sified terminal with the Baseline System. As in the Baseline 
System, if a user would want a classified entry it could be 
ordered for delivery by mail. 

IV. QUESTIONNAIRE 

We plan to visit or phone you for a personal interview at 
which time you will be asked to answer the questions orally. 

The purpose of this questionnaire is to give you some indica¬ 
tion of the scope of the questions and to give you time to 
think about the answers. 

1. Please identify yourself as: 

a. Information Specialist (e.g., librarian) 

b. Researcher 

c. Research manager 

2. Estimate your use of DROLS over the last year 

a. Daily 

b. Weekly 

c. Monthly 

d. Yearly 

e. Never 
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• 3 . 


What proportion of the documents (not books) in your 
collection is classified? 


4. Have you used commercial on-line bibliographic services? 

a. Yes 

b. No 


5 . If you have only unclassified terminals, why do you not have 
a classified terminal? (Use as many answers as apply.) 

^ a. No need for classified information 

b. Little need for classified information 

c. Cost 

d. Terminal procurement difficulties 

e. Secure facility difficulties 

f. Necessary information is available in other ways 

g. Other (comment) 

• 6 . If you have a classified terminal, on what basis do you 

justify the additional expense and provisions over that 
for an unclassified terminal? (Use as many answers as 
apply.) 


a. Necessity for classified information 

b. Response speed for classified information 

c. Cost not a problem 

d. Terminal provisions not a problem 

e. Other (comment) 


7 . Would you consider changing your terminal to unclassified 
given the changes in DROLS programmed for 1980 ? 

a. Yes 
a. No 


8 . How was the capital (equipment) expense of your terminal(s) 
funded? 


9. 

10 , 
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12 


a. By our library 

b. By our organization (not library) 

c. Contract or Project 

d. Other (How?) 

Do you know how much It cost Installed? 

How Is the operational expense of your terminal(s) funded? 

a. By our library 

b. 'By our organization (not library) 

c. Contract or Project 

d. Other (How?) 

Do you know how much it costs? 

If only unclassified service were provided, what would you 
most likely do about the missing classified fields (some 
titles, descriptors, identifiers, or abstracts)? 

a. Nothing 

b. Order the classified entries 

c. Order the classified report 

Referring to Question 12, on what basis would you take 
actions 12a, 12b, or 12c? 

a. Ignore the report; inadequate information and too 
much delay in other options 

b. Have enough information; no more needed 

c. Require entry information to proceed 

d. Might as well order report 

e. Other (comment) 

14. Would you be satisfied with an unclassified DROLS? 

a. Strong yes 

b. Moderate yes 

c. Indifferent 

d. Moderate no 

e. Strong no 


13, 
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15- Why? 


a. No strong need for classified bibliographic information 

b. Need classified information, but can function adequately 
without classified DROLS 

c. Cost of classified terminal is too high 

d. Can use DTIC searches 

e. Difficult to provide for classified terminal 

f. Strong need for classified information 

g. Classified search faster with classified terminal 

h. Search more fruitful when performed in-house rather 
than by DTIC 


16 . 


What modification to DROLS (not discussed heretofore) do 
you think would be valuable? 


C-ll 


